Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix example int8_inference_huggingface.py #414

Merged
merged 2 commits into from
Feb 27, 2024
Merged

Fix example int8_inference_huggingface.py #414

merged 2 commits into from
Feb 27, 2024

Conversation

alexrs
Copy link
Contributor

@alexrs alexrs commented May 19, 2023

What

When running the example int8_inference_huggingface.py I was getting the error:

ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported

when using the latest transformers=4.29.2. According to this issue:

the tokenizer in the config on the hub points to LLaMATokenizer. However, the tokenizer in the library is LlamaTokenizer.
This is likely due to the configuration files being created before the final PR was merged in.

(See comment)

Fix

Use LlamaForCausalLM and LlamaTokenizer instead of Auto... classes works as expected.

Test plan

  • Run python examples/int8_inference_huggingface.py
  • Obtained the following output:
Hamburg is in which country?
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.
The capital of Germany is Berlin.

@USBhost
Copy link

USBhost commented May 23, 2023

decapoda-research is extremely out of date. Use a newer convert.

@github-actions GitHub Actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

@github-actions github-actions bot closed this Dec 30, 2023
@TimDettmers TimDettmers reopened this Jan 1, 2024
@TimDettmers
Copy link
Collaborator

I think we should change the example model used instead of fixed the issue with the tokenizers. @younesbelkada any recommendation which model we should use?

@TimDettmers TimDettmers added high priority (first issues that will be worked on) Low Risk Risk of bugs in transformers and other libraries labels Jan 1, 2024
younesbelkada
younesbelkada reviewed Jan 25, 2024
View reviewed changes
Copy link
Collaborator

@younesbelkada younesbelkada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing ! I left one comment

examples/int8_inference_huggingface.py Outdated Show resolved Hide resolved
younesbelkada
younesbelkada approved these changes Jan 25, 2024
View reviewed changes
Copy link
Collaborator

@younesbelkada younesbelkada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for your contribution !

@Titus-von-Koeller Titus-von-Koeller merged commit 4b232ed into bitsandbytes-foundation:main Feb 27, 2024
1 check passed
@Titus-von-Koeller
Copy link
Collaborator

Thanks a lot @alexrs for your work and @younesbelkada for your work.

Sorry that this took a while to merge :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@younesbelkada younesbelkada younesbelkada approved these changes

Assignees

@younesbelkada younesbelkada

Labels
high priority (first issues that will be worked on) Low Risk Risk of bugs in transformers and other libraries
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@alexrs @USBhost @TimDettmers @Titus-von-Koeller @younesbelkada